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Glossary of Ke ^ ' Terms 



^ 




Evaluability assessment - A process for determining in advance' the like- ^ 
lihood of an evaluation's' success. It consist's of two stages: first^ 
an examination of program characteristics and second, an evaluation 
^feasibility Analysis. See. the entry for Rutman, L.^ Planning useful 
evaluations . 

Evaluation - . The "process that involves (a) posing questions about the Wur- 

pose, implementation and consequences of ^ \ , programs, and (b) systema- 
tically collecting and analyzing data confcerning those questiensw where 
both of these jactivities a^re intended to 'facilitate judgment aboiit the 
worth of such programs*' (Weiner, Rubin> and Sachffe, p.l). An ^valuation 
report , often re^ferred to as an evaluations is the product of 
evaluation. ^ ^ 

Modes of use- Using various labels,, theorists h^ve distinguishedy'three tnodes 

of use for evaluation results: instrumental , conceptual , ana symbolic 
, * (See Caplan et al^. , 1975; Knorr, 1977; Rich, 1977; Ueiner, /Rubin, & 
Sachse, 1977; Weiss, 1977; and Pelz, 1978). 

Instrumental or allocative use refers to a case vher^ an action is 
taken in direct- response .to the results, of an evaluation/ Implicit in 
the term is an input-output model of organizational behan/ioij; either the 
results suggest a needed change and the decision^maker Aakes it, or they 
provide support for^the status quo and no change is needed. 

In. contrast, ' conceptual or appreciative use refers to cases where 
evaluation results influenlce decision-makers' current/ thinking a\out 
\ (and poterttially their ^future action regarding) an issue or program. 
Rich (1977) labels this- 'Tcnowledge for understa^din^^' as opposed to the* 
> insirrumental ''knowledge for action." 

A third type of use has little to do with .the/'^tual content of 
the evaluat;ion results.' S ymbolic or legitimative/use refers to cases 
where evaluation results are used indirectly for/a variety of purposes, 
for example, to garner political support, to substitute for a decision, 
or to discredit a disliked policy. Khowing thair desired action^ decision- 
makers may seize on the results and manipulate/ thetn to a desired end. 
Clearly> any given, evaluation* can be used in all of these ways, especial- 
ly in ac tive^^ettings where there are numerous decision-makers and infor- 
mation users. 



Personal facto r - The presence of at least one 
hfho cares about the process and using its 
Patton, M.Q» et al. > In search of impact. 



irson in an evaluation stutfy 
results. See the entry fQT 
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Utilization - An instance of utilization occurs when evaluation information 

is considered by someone as arf influence, in ''making decisions, substanti- 
ating previous decisions or actions^ or establishing or altering attitudes" 
about any of a variety of items (adapted from Alkin, Daillak & White, 
p. 232). Two vieys of utilization exist currentlyr*a mainstream view 
that looks for the specific impact of an evaluation on subsequent decisions 
and an alternative view that examines the numerous direct and indirect 



effects an evaluation can have on an organization (See "Modes of use^' 
above), Weiss (197?) suggests that the term utilization be dropped in 
' ^ favcrr of the simpler and equally apt term use , ^ . 

Utilization - focused evaluation - An evaluation designed to insure that. its' 
results will be put to use; As discussed by M,Q, Fatton, there are two 
essential requirements: first* identifying and involving relevant decis^ 
makers arid information users; and second** being "^ctiMe> reactive and 
adaptiye*'^in working with these users. See. the entry for Patton> M.Q., 
Utilization-focused evaluation. 
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An iroportant garlyT)iece . . " ^ 

■ ; ^ ' '■ \ ' - . ■ ^ 

Weiss, C. H. Utilization of evaluation: Toward comparative study. In' 
C. H. W^^lss (Ed,), Evaluating action programs; ' Readings In social a^ion and J 
education . Boston: Allyn and Bacoo> Inc*, *1972, 318-326. ' - ' 

ift evaluation results are not used to make, program decisions, Weiss writes, 
then evaluation has ''failed in it& major purpose." Two types, of limitations 
lead to the non-utilization of results: first, fhe numerous and complex organ- 
izational factors that tend to work against the implementation^ of results ; and 
secondj the then current state of evaluation practice. 

This seminal papers originally presented in 196&, proposes empirical study 
of evaluation to enhance the ticilization oi results. ,To Weiss^ .research on 
evaluation should examine tljree majbr types of use: use^ within ongoing programs 
(what we would now call formative evaluation); .use at the completion of 
programming cycles (summative evaluation); and use "^n outside agencies. Weiss 
suggests potential and testable ways for conducting evaluations to enhance their 
utilization: first by explicating the theoretical framework of programs; seconds 
by' creating a "process model" of programs; and third, by Analyzing the effective- 
ness of specific progf^ra components, rather thatr^programs as a whole.. Other areas 
she labels for study include the targeting of evaluation results on, potential 
usars, the involvement of administrators and, other prpgrara pairticipants in^the 
evaluation, the timely cojnpletion and release of evaluation resultSj and jsffpctive 
n^thods of communicating and disseminating r^esults. " ■ . 
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Studies of evaluation uEilization ^ 

: — ^ » -* 

Alkln, M. Daillakj j & Wbite> P» Using evaluations: Does evaluation 
make a difference ? Bevjerly Hill^: Sage Publications j 1979. 

AlW^nj Daillakj and White distinguish two views o£ evaluation utilizationr 
a mainstream view thaf looks for the specific impact oi an evaluation on sub-*- 
sequent decision^; and a broAderj alternative view that examines the numerous 
direct and indirect effects an evaluation can have on ^n organization. The 
consensus "that evaluation results are not being used may_be due to the bias of 
the mainstream Iperspective; in reality^ Evaluation results may already be in- 
fluencing programSj but in more subtle ways than the search for static factors 
affecting utilization would .suggest. 

The case studies presented here examine th^/^interaction of people and 
situations in the evaluation process '' (p. 26j italics in' original) , Using a 
naturalistic research strate^yj Alkin and hi-s colleagues pieced together case 
studies of five completed evaluations. Participants in the evaluations were 
interviewedj in some cases several times,^ apd they then, read and corrected the 
finishe^deseriptions , Also included were the participants' fin^i reactions 
and responses to the experience. , Although in only one case was the utilization 
conscmant with the mainstream definitions evaluation utilization did in fact 
0C9fur in each casej lending support to the alternative view of utilizationi 

In their final chapter^ Alkin^ DaiHak> and White summarize support for 
this broaderj alternative definition of utilization having demonstrated that the 

re of the mainstream approach inherently limits the results. They name four 
\ . . . ■ . 

essential components of AJtiiization — 1) eyaluation^information mdfst be communicated; 

m 

2) an ''appropriate user*' must be evident; 3) the irrforraation must, be labeled 
a single input or one of several; and 4) the information must be used in agme 
way — and define an ''instance of the utilization ol local school program evaluation" 
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Evaluation information considered by a local client* 
sanctioned local users> or external users' as a dominant ■ 
ihf]jUence> one of multiple influences* or one of ' 
multiple* cumui^ative influences in making decisions* - ^ 

substantiating previous decisions or actions* or 
establishing -or altering attitudes related to establish- 
ment > external funding* local district funding* 
continuance o^ a component* curriculum/instructional 
methods* administrative/personnel operations* or* 
community acceptance of the local school program (p,*232')* 

Clearly certain characteristics of evaluation sitUatiotis are associated 

with the Utilization of results; and* based on the'ir x:ase study results* Aikiiv 

et al, develop an analytic frame^^sprk consisting of the following eight- factors: 

preexisting evaluation bounds; orientaVion of 'the users; evaluator's approach; 

evaluator's credibility; orgaoiza^t^onai fact or? T extraorganizational factors; 

information content and report^^; and administrator style. They point to tbe 

need for further focused studies in the development of a theory of evaluation 

utilization. 
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Lyon* C. D.j Doscber* L, j McGranahaOj P,j and Williams* EvaluaElbn 
and school dl'strlcts , , Los Angeles: Center for the Study of Evaluations 1978. ; 

The Evaluation anli School Districts Proje^it at- UCLA's Center for the Study 
of Evaluation was designed ca "identify and. analyze educational, evaluation at 
the local educational agency (LEA) level" (p. 1). This report presents the 
results of a 1978 survey of evaluation characteristics and practices in over 
200 districts that^ enroll 10,000 or more students and that have an organizational, 
unit formally responsible for program evaluation. ^Iso reported are^the results 
of related fieldwork. ' ' ' - 

Many of the results have implications for the utilization of evaluations in 

public' schools . Evaluation in these large districts is an in^house activity. 

School district personnels rather than external consul^tants * do the .major share 

of evaluation office workj and contrary to popular belief* these units are primarily 

supported by local fundSj not federal or state monies. Although little agree- 

ment ^xists on what constitiftes basic evaluation practice or on what evaluation' 

activities deserve ,priorityj evaluation in these units freq^uently means achievement 

testing; roughly 75% say that '^student achievement Is the dominant topic of 

data collection" (p. 76); and an equal percentage say that "testing is their 

major method of data collection" (p. 79). 

* 

The relation of such evaluation to improved instruction Is tenuouSj espe(H:lally' 
given the organizational position of many units. Rather than reporting directly, 
to the'Superintendent* evaluation units are more likely to be in one of the 
typical lines of authority (e.g. Instruction or Administration); however, 62% of 

* m 

the offices are not located in .the Instructional line. Development activities 
in these offices generally center on tests and evaluation irfstruments > rather than 
on instructional programs^ and products. Ttiis is partly explained by looking at 
the clientele of the average unit; roughly 60% of the time is spent wixh 



administrators compared to only 40% with instructional clients* Reflecting 
these results/ one of the propositions fo be j^nvestigated in further >r<^ject 
' research Is that "/'evaluation of flees, are not ©s^atly Involved injjianaglng ^ 
educational actiMties*' (p. 32)\ ' ^ 

Two thirds of the evaluation heads responding feel that their personnel 
resources are inadequate, and, asked* what^ would Improve their units* effectlve- 
^nessj'they generally Identify !*a<iditional staff," "increased Access to computer 
time and programs," and "information about effect i;ve school district jevaluatlon 
^practices/' rather than, for example^ organizational changes or Increased 
communications (p . 112 > 1 14) • Interest ingly> most respondents report Iqw- 
ambiguous-, low-conflict experiences in their work {p, 115) despite ^thelr j^resumed 
involvement In highly political contexts, , ' 
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, Meltsnej^ A, J, Policy Analysts jn the Bureaucracy . ^Berkeley; Unive;:sity 
^L of CaUforflla Press* 1^76i, 

. Based on interviews With 116 federal policy analysts, Meltsner conceptui'lizps 
a typology of evaluators* He .argued that e-valuators ±a.y be differentiated 
primarily by how much political and a^^alytlcal skill they have. * 

Political skill refers to the ability to exerjnise influence over various 
foras of decision-makingt '^Entrepreneurs" are exceptionally able on both'skill 
t- dimensions. "Techpicians" have good analytical skill but less political skill* ^ 
"Politicians" have good political strive but their technical skills are below 
average. . ^'Pretenders" are weak on both skill dimensions, * * 

This typology may correspond po stereotypes ^which both administlrators and 

evaluators ^employ in predicting or understanding behavior. Thus j if several 

/ ^ ■ . * ■ " . ^ ^ ' 

theorists are correct in^rguing that the personal f^tor Is a primary detertninar\t 



oj^ use, this typology may par£l)^ explain important dynamics of evaluator-adyilni^trator 
Interaction. Meltsher also of^ers a typology of clients. His workalso includes^ 



recommendations for optirflzing'evaluation use; for example, he tecommend^ that 
■ evaluatgrs ^target evaluations toward speci fic administrators, This work Is siiailar. 
^to'Patton*s {1978) In flavor* arfd complements related worjcs which have examined 
^ use In related evaluation areas such as health programs or local educational * 
agencies , ^ 
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Newnian> D. L*,^BrownrR* &*Bra^kamp, U, A, Communication theory and ^ 
the utilization of evaluation. In Braskamp, L. A/ & Browfi, R* D. -(Eds.), 
Utilization of evaluation information . £an Francisco; JosseJ-Basfej -1980.. 

This review aVticlTe summarizes a "five-year series of studies which has 

systematically^ examined the relationship among the characteristics of^an 

eyaluator, an evaluation reportj evaluation audience characteristics, 'and 

audience responses'^ (p. 30). Braskamp^, Brown, Newman, and pther associates have 

asked subjects, near}.y LfiOO in all, represjentlng a va-riety of evaluation 

auidences, to Yfead and fespond to simulated evaluation reports* Aj^though the 
*■ ' ' ' * 

generalH-zability of such studies "can" be question^, the res4ilr*s^ suggest Several. 

importa-nt points concei^ning the utilization of evaluation results. 

First, _,both the title and sex of the ev^luator can affect audience reactions,. " 

Even though readers rea<I^ii(entical reports, they rated those they thought wr^.tten 

by a "researcher** as significantly^ more objective than* those written by an 
* . . * ' " 

"evaluator" or a "content (art) specialist/' Report readejrs were- les^ likely 

- ' f . ^ 

.1 ■ , . ■ ■ * 

to agree with reports wfeich they thought \fere writtjen.by female evaluators wh^n , 

the tiel-d diffiered from their own; in their o\m ^ieldy they .were less critical 

of'^the female's results, although they were still moxe likely to agree with the 

male evaluator's recommendations. * , - ^ 

Second, fehe use of Jargon and data can affects audience 'ratings Of technicality 
* * ■ " ■ V ^ ' * 

and difficulty. The report rated inofit difficult by readers included jargon, but 

no data Support, *''Generally, reports containing both jargon and data were rat^d 

more liseiul regardless of whether the readers were ^rof essionaTs in the field 

or were lay persons from another field*' (p- 33)* Rated next highest vere reports, 

lacking both jargon and data u$e, suggesting an interact!Uon of the use or non- 

use of jrargontand data in the reactions of readers. The -type of information included 

in reports also affected decision-makers* 



y ■ ■ , * '10 . 

Third, audienc^e characteristics can make a difference in reader reactions. 

Ratings of the usefulness of the results of arr external evaluation differed 

depending on the organizational ^>o&ition of the audience". Other important variables 

w^re Jthe leyel of prof es^sional experience and the field pf the reader. Also 

involved is* the^ "audience's perceived need for evaluative information in a 

particular area!' (p* 33). 

The authors suggest the ne^d for continued research in the area of communication 

> 

and attribution thfeory, including the effect of it>formal evaluation reports on . * 
decision-makers and the 4egree of Audience understanding of the material being 
assessed. ^ They point further to the nee'd for similar observations in more ^ 
natural settings. ' , 
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Patton>>M. Grimes^ P. S.> Guthrie^ IC/M"; Brennan> N. J,, Greneh>vB. D. 
and Blythj D. a. In search of impact: ,An analyses of the utilization of federal 
health evaluation^ research. In T. D. CooK^and Associates (Eds.)> Evaluation 
stiidies revlev annual > Volume 3. Beverly Tiills T Sage Publicationfis> 1978'*.^ 

This chapter summarizes the follow-im^study ?atton and his^s^o^iates 
conducted of 20 federal he^^th program evaluations. For each evalua*tfen>,ijtwo 
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key figures»were interviewed: the evaluator with major responsibility for th 

evaluation; and the person selected by the project officer as the--key ^deglsio^T 

maker. 'Open-ended questions allowed the respondents *t;o define anStf^iscuss 

utilizatioir in "p^i^sonally meaningful tenns^" They were then asked to comment 

on the importance of 11 , factors which the literature suggested should affect/*^ 

utilization (methodological quality^ methodological appropriateness* timeline^^s., 

lateness of report* positive/negative findings* surprise of f indings|lcentral/ 

peripheral program objectives evaluated* presence/absence of relateostudies > g 
* ' * 

political factors, government/evaluation, interactions* and resources available 
for the study) . , ' . ' ' 

In contrast to the popular feeling that evaluation results are underutilized, 
Patton et al. found- that results were in fact being used, but. no t^ in the general, 
sense of having an "immediate and concrete effect on specific decisions and * 
program activities" (p- 143). Instead, evaluation resyits provided the decision- 
maleers one addij:ional piece of information, " thereby permitting some reduction 
in the ^uncertainty within which any federal decisionmaker inevitably operates " 
(p. 145, italic? in original)* 

Evaluator^ and decision-makers identified as important to the utilisation. ' 
'process only one factor from the list of 11 suggested by the literature: a . 
political considerations factor, as might be expected. The ten remaining factors 
were not ipentioned consistently by the respondents, and the importance of these 



yariablesi. for u^jULza^ion was thus into question. The only 9ther factor 

identified by these practitioners as important was a factor not on the original ^ 
list J one which ^atton and his" associates labeled^ the personal factor^ The 
personal factor eme^ed from the interviews as the frequently repeated comment 
that ill cases Wher^ evaluation ,utili2atioa occurred-, it was largely due to the 
presence of a person who car$d about tbe evaluation and its results; without 
the^personal factor, underutilization could he expected, ' 

The isiplications, of th^se results for evalnator-s are striking; to see that 
results are used> evaluators need to consider Jboth .the poli*tical context of the 
evaluations they conduct arid the need for the one person who will make things 
happen, both during atid 'after th^ evaluation. Equally striking are the implications 
f or'ev^luation resear<?hers; who first need to reconsider tV\e narrow definition of 
Utilization which ignores political realities real-world decision-making and 
who also need to'study tlie newly identified personal factor with care. 



Several re^eTXt'doctoral" dissertations have ^addressed the question of evalua- 
-tion utilization. * ' ^ ^ ^ - . ^' 

' . 1. Carlson'j P, .The relationship of three, organizational variables on ^ , 
th^ utilization of evaluation^ (Doctoral dissertation, Ohio State University, 
1973). , Dissertation Abstracts Infiematlonal ^, "1974, $4 7097A (University 
Microfilms No, -74 »- 10, 924). 

» ' ■ ^ * * 

Carlson studied the relationship of utilization, as measured by a - . ^ 

self-developed iadex, and three variables: the clarity of organizational goals 
and object^ives; the number of indiyicfuals necessary for approval of a recom- 
mendation; anid the status of the evaluator. Positive relationships existed 
between utilization and cT&rity'and between utilization and the internal 
states, of the evaluator. No such relationship was found between utilization 
and the number of individuals needed for approval* A factor analy&is of the 
utilization index suggested the existence of three factors; reliance^on / \, 
traditional fdrms of evaluation (product evaluation) ; .reliance on previcJ^s 
experience, intuition, or familiar materials; and negativism toward evaluation* 

2. GranVllle; 'A,C. Where do school decisions realty cotne frm? 
(Doctoral ^dissertation^ University of California at lio^ Angeles, 1977), Dis- 
sertation Abstracts International , 1977, 38, 743A (University Microfilms 
No. 77-J6, 166). ' \ 

In* an empirical study, Granv±lle asked 157 elementary and secondary 

J* 

principals to read evaluation reports differing on thre^ yariables*; the informa- 
tion soj^rce, the suggested imf>lications for adoption, and the type of program. 
He th^n measured their incHn^^ation to adopt the program described and collected 
naturalistic data on 14 other principals. The following conclusions were'reached 
* (1) In addition to objective evidence, social an3 political factors 
Influence organizational decisions; , ^ * ^ 

(2) Objective factors are more influential under unusual conditions; 

(3) Decision-makers in orgai^zations of different sizes are differential 
ly ffespousive to influence; 

(4) f Tha social influence of people is in liart determined by the degree 



to which decisions will affect them; and 

^ , (5) Individuals* political influence iS determined in part by the^ 

extent to which they control resource allocation/ ^ 

\ ■ - 

3. Andrews/ J. V, Reactions to program evaluation: A qualitajilve analysis 
(Doctoral dissertation', Cornell University, 1979). Dissertation Abstracts ^ 
Internationals 1979, 39, 6717A (tJniversity Microfilms No. 79^10, 731).^ ^ _ 

' ! ^ . 

^ * "1 

* Andrews' sii case studies of recent evaluai;ions included 27 participant . , 

interviews* Nine themes for future research were extracted from these inter- 

views/ including the question of the sizable impact of prior knowledge and 

beliefs on evaluation, of the importance program personnel attach to being / 

participants, and of one*s role in the organization. ' 

> 4. Dickey, B.- Utilization of evaluations of small scale, educational 
projects (Doctoral dissertation. University of MiAnesota, 1979). Dissertatitin 
Abstracts International , 11979, 40, 3245A ('University Microfilms No. 79-26, 
115). ' . ' ■ 

To determine factors affecting the utilization of A7-^ompleted Title IVC 
evaluations, Pickey interviewed project directoi^s^^^read final evaluation reports, 
and^collected archival data. She concluded that the likely explanation of 
underutilization lies in factors related tp the natural resistance to change 
and to the dissimilarity of the academic and real worlds, rather than to a 
technically impoverished state of evaluation art. (See also Dickey, B* , Utili- 
zation of evaluations of small'-dcale inrvDvative educational projects. Educational 
Evaluation and Policy Analys;ts , 2^,6, 1980 , ,65-77) . 

5. Gray,^.J. The development and-^testing. of a collaborative approach 
to education program evaluation (Doctoral di3sertation. University ofr Oregon, 
^1979). Dissertation Abstracts International , 1980, A0> A966A (University 
' Microfilms No. 80-5, 767). 

In his dissertation Gray developed a coli^oratiVe approach to program 

J 

^ f 1 

evaluation -involving the evaluator with evaluation audiences. The approach 
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was reviewed by a gradtiate clas^ and practicum,'by public school personnel, 
and by evaluation/ specialists . 



6, Weeks, E*C, Factors affecting the utilization of evaluation findings 
in administrative decision-making (Doctoral dissertation. University of 
California, Irvine, 1979). pissercation Abstracts International , -1979, 40, 

457A (University'Microfilm^ No/79-15, 9201" 

I 

This study sought to i/^dentify which of three variables correlated most 
highly Vith utilization. ^^The variables studied were the organizational location 
of the evaluator; the dec ts ion-making context; and the nwtthQdological practices 
employed. The negative correlation between research design and utilization 
suggests thaX decision-makers have a slight preference for more 'qualitative 
forms of data' analysis . ^ % . ' - , . ' 



be effectively utilized'' (p, 9), 



Praccical ^discussions of evaluation utilization , I 

Ha^n, J, .*F,._ Reasons vhy eval\iationg a^nd testing don't inform . A "paper 
presente^^ at^ fl^annual meeting of the Americ^ Educational Research Association, 
Bostoni!;i98p, , - ^ , ' ' - 

i 
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I I » ' 

, ■ ^lte^i,ew±ng the literaturej Haenn discusses three' sets o^ factors which inhibit 
infon^ation u^e*^ The first are organization characteristics; the structure, 
cliiaatej and politics of organizations may limit the effective utilizatioa of ' 
infojrmation* A second type* inhibiting factors are the personal characteristics 
bfrusersj whose information needs* interests^ and abilities may affect the use 
ppf information. The th^,rd set of factors encompasses methodological characteristics 
of evaluations arii reporting, including both the role of the Evaluation aitd the 
characteristicis of its reporting* Haenn summajrlzes his review of the literature 
by noting that **the literature is filled with reasons why information may not 

/ 

Following discussion of lihese factors, Haenn presents a similar model of local 
school district influences on the use of evaluation an<^ testing information. 
Organizational characteristics are divided into those that are relativejly static 
(e,g, the size and complexity of the organization) and those t^iat are more 
easily modified (e,g, inte'rpersonal characteristics and evaluation credibility) < 
Personal characteristics subject Co modification include use^s* attitudes toward. ^ 
evaluation, their problem solving capability, and their cosinitment to use< ^The 
Tueahodological characteristics of evaluation and especially of evaluation reporting 
are also part of the influence model- ^ 

Having presented potential reasons why evaluation and testing do not inform, 

Haenn then gives three strategies for facilitating their use. Fir5t creating 

dem£fnd — for^example^ through "establishing a cliinate reflecting the value of 

evaluation dnd testing information" (p; 12) or through employee training 'techniques- 

* ^ 
helps insure .that users will welcome evaluation infprmation^ Second, increased 

cooperation between the .producers and users of eA^aluations can. have a Similar 



effect. This cooperatioT\ can be encouraged, for example, by providing technical 

assistance or by increasing interpersonal comm^^iic^tion. .Thirdj improving 

^ \ ^ [l - ' . 

repotting procedures can also facilitate th^/us^ 6f evaluation results. 
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Three papers by F. M. Holley and her associates' 

^ 1. Lee, A. M. £r Holley, F. M. Comiflanlcatlng evaloatlon Information:^ Some 
practical tips'that work , A paper presented at the annual meeting of the American 
^ Educational Research Association, Toronto, 1978.- 

' ' t' ' ' 

Iq this highly readable paper, Ann Moore Lee and Freda Holley give advice 

on how to 'disseminate evaluation results. Their practical communication principles' 

cover^six topics: evaluation audiences; the evaluation message; the writte^ 

medium; verbal prese^ntations; difficult audiences; and working with the press. - 

-f , » ^ 

Included among the tips are such things. as relating the evaluation information 

to action which must be taken, starting reports with the most important information, 

and training the pr^ss to properly iijterpret evaluation data. 

2. Holley, Catch a falling star: Promoting the uti'ligation of research 
and evaluation findings . A paper presented at thfe annual meeting of the American 
Edtic^ational Research Association, San Francisco , 1 979 . 

Wri^ten^on a more theoretical lev^l, ,this paper has as it© premise that 
researchers and evaltiators must accept responsibility for promoting the utilization 

of their results. Holley presents the following franlework of utilization factors: 

\ ' ^ ^ 

characDeristics of the thing evaluated; characteristics of the evaluation user; ^ 

. characteristics of the organization; characteristics of the evaluator; characteristics^ 
of the evaluation findings; and dissemination resourced available to the evaluator*. 
Then, using an imaginary case study* she describes how these factors affect the * 
behavior ijf ap ^^aluator conducting an evajluation, ending happily in the results' 

''being used. By taking concrete action toward utilizations she writes* evaluators- 
can help improve the use of evaluation information. ^' 

3. Holleyj F. Evaluation utilization: Is it easier t:o move a mountain 
than a molehill ? A paper presented at the annual meeting of the American 
Educational Research Association, Boston, 1980. 

The'major claim in this paper is that the potential utilization at any 



19 



evaluation is Heavily influenced "by the pol'itical context in which it is 
conducted, and that, knowing that^ evaluators should u$e .potential utilization as 
a key criterion in the allocationof evaluation dollars. In other words> 
evaluators should tackle district mountains^ which have a; high probability of 
b^ing moved and ignore more interesting molehills, unless and until they too. 
] grow into peaks. To^ support this claim^ Hollay discusses an example of the 

utilization o^ a staff development program in .the Austin Independent School District^ 
noting the tremendous -^amount of effort required to get evaluation results -used* 



/ 



ERIC 



23 



Pattonj Q. Uttltzatton-fociised evalu^ton . Beverly Hills: Sage 
Publications, sl978., 

^ Patton's book is b. reaction to the widely accepted conclusion that evaluation 
results are un-- or at .least under-utilized. The implicit definition In earlier 
wbrk on utilization^ required an "immedi^ate » concretes and observable "effect on 
specific decisions and program activities" (p. 24). This definition is 
inappropriate^ whites Patton, since evaluation research is merely one piece of 
information affecting the development of programs in actual settings. Smpmariz^/ng . 
an earlier empirical study in which he participated (Paj^ton et al,j 1975), he 
notes that in that study, 78% of ,the decision-maket^interviewed and 90% of the 
evaluators .felt that evaluation had had some effect; on the examined programs/ 
Evaluations are utilized, but not necessai^ly in ways/ evident to researchers 
seeking obvious and direct effects. ^ " ■ ! 

* 

Recognizing that evaluations are inherently political, utilization-focused 
evaluation has as its central contiern what an evaluator caa to;-insure that . 
evaluation results will be used/ There are two essential requirements: first* 
the relevant de9ision-makers and information ^^^^ must^ be identified and take 
an active part in the eva^luation; and second, the evaluators must be "active^ 
reactive^", and adaptive" in working with *these users; The "personal factor" 
identified i-ti ^atton's earlier work pointed to a yasic principle ''of evaluation 
iitilization: where at least one person with authority cat^s about the evaluation* 
the results are more likely to be used* For tiiis reason, utilization-focused ^ 
evaluation uses an interactive process involving this person (or persons) and the 
evaluator^. " Focusing the ^valuation question, establishing goals, and selecting 
de^sign and'data collection methods — all involve the evaluator working collaboratively 
with the decision-maker/inf ormat ion user, \ - ^ ^ 

The utilization-focused evaluator helps to establish and then test the causal 



model upon which the program is based, using an '"tnd'uctive, pragmatic,^ and ' 
highly concrete'^ approach. Data that Imve high face validity for the usets 
are collected, and the results should ftot be surprising because of th^ 
collaborative nature of the entire evaluation process. The value of the 
results of a utilization-focused'evaluation lies in their usefulness, i,e. 
the extent to which-they tell the decision-maker what to do next. Given tJiis , 
the results will have their, desired impact/ 



^ Rutman, L. Planning useful evaluations . 'Beverly Hills:' Sage Publications', 
1980. ' . ' ' , ' ' ■ 

The topic ot Rutman's book is. evaluability assessment, a process for 

' . ' ' ' / ^ * ' ' ^ . , 

deteTminiiig^in advance the likelihood of an evaluation's . success ♦ Developed 

inittally^by Joseph Wholey and associates at the Urban Institute, evaluability' 

assessment focuses first pn ^program characteristics and tth*en on- the feasibility 

of conducting an Evaluation study as planned*: ' \ ' 

An evalu^bjLlity assessment -begins by examjLng characteristics of program 

components to^ determine how close they comife to the ide^^l, asking, if they axe ' ^ 

■J " '-^ - 

well-defi^ned ana Capable of being impl*emented in a prescribed manner^" if goals 

Snd effects are clearly specified; and if plausible causal connections link goals 

and effects- 'In this stage, the jevaluator deyelops. t^hree models of the progtant: 

first J "a. "program documents jaodel/' which shows the cas.ual links described in 

program materials; second* a "progtam mina^er',s model>^" which modifies the ffrst 

model according to inf<ynnation from key decision makers; and> finall^^ an " * * 

"evaluable. program model," which presents the evaluator's views of 'what> components 
\ - 

can appropriately be evaluated- ' . * 

\ * * 

Because the purpose of the evaluation determines its methodological require- 

ments, the second stage of the evaluability assessment* the feasibility analysis, 

begins .by determining the ^purpose(s> of the given evaluation; then looks at the 

constraints oh the evaluation to see to what extent 'the , research requirements 

can*be met* Considered in the feasibility analysis are program design and 

ii^^em^ntation> information requirements, and research* design. Its product is 

a list, based on the evalu-able>'program model, of the program components and the 

goals and effects to be. studied in the ultimate evaluation* The program evaluation 

whfiTi it is finally conducted, benefits from the limits set during the J^||Kstages 

of the evaluability assessment; only what can and should be evaluated will be 



evaluated < 

- Programs themselves can also benefit from evaluability assessment. Before 
the program evaluation begins and as ^ by^proauct ofthe evaluability assessment 
process, program managers may make' changes in the program to enhance Itfe 
evaluability, Str^egies may incXude analyzing problems ; specifying outcomes; 

- ' / - - ' ■' 

--assessing program design and implementation; and facilitating ptogr^ development 

(what Rutman c^lls ^'formative research")* 

^ The benefits of conducting evaluability assessments, then, fall into two 

categories. First, evaluability ^ses^ments facilitate evaluation planning by 

establishing priorities* by providing "frdnt-end control'* over^the evaluation 

process, and by allowitig a wise^llocation of evaluation dollars. Second, they 

facilitate planning by^ providing information on appropriate , directions for , 
* 

program manager^, ' 



Theoretical suggestions . - - 

Conner, R.F, Measuring the utilization of evaluation fimUngs ; De&cription 
. and critique of different techniques . Unprinted mimeOj University of Cali*- 
fomia, Irvine, 1980. ' * ■ * 

In this revision of a 1979 Evaluation Research Society paper^ Conner 

discusses the research metHodology used in six utilization studies; Weiss 

" and Bucuvalas (1977 and forthcoming); Knorr (1977); Patton et al. (1975); 
Alkia, Daillak, and white (1979); Heiss (197A); and Burt, Fi«k, and Hatry 
(1972). He proposes a four part evaluation model consisting of goals, inputs 
processes^ and_ outcomes, then makes comparisons using the categories of basis 
of judgments, time orientation of judgments**, subjects, measurement method, 
definition of use, and foci of study. * 

Conner concludes that more studies of the ongoing (as. opposed to the 

* retrospective) process of evaluation utilization are needed, ''along with 
studies involving higher level decision-makers. He further suggests both 
more systematic quantitative study of utilization and continued* qualitative 
work. Finally, he feels that utilization goals and inputs deserve more care- 
ful attention* than they have yet received* 
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■w4iner, S*S., "Rubin, D., & Sachse, T. '^ Pathology In instttuttonal structures 
for evaluation and ^a possible cure ^ Stanford, CA: Stanford Evaluation 
Consortium, "f9^/ 

Weiner, Rrfbln, and Sachse write that two reasons are generally given 
to explain yijy evaluation results are underused* The first explanation^, is 
that fhe results are of low quality, suggesting a need for an improved tech- 
nical oj: scientific evaluation crarft* In recent years, the wisdom of thlfe 
claim, with, its reliance on an assumed organizational rationality, has b6en 
called into question. * The. second explanation, sometimes called the Two Com- 
munities Theory, is that policy makers and evalaators live in worlds so dif- 
ferent that th^ results of social science style evaluations can have little 
impact In the real world of decision-making. Discussing governmental contracted 
evaluations, this paper supports the second notion, but gives a related and, 
in a sense, more basic reason for the failure of evaluation/ 

Rather than trying to suggest^ change in the behavior of selected ^ 

0 

policy makers and evaluators, Weiner, Rubin, and Sachse suggest that the 
* 

structure and r^ations of the institutions involved need to be changed, fol- 
lowing two unusual examples of useful evaluations, they discuss three sets of 
constraints — organizational constraints* political constraints, and thfe *'pre- 
vailing views of professionally legitimate acti^Jity" — in addition to assumed 
constraints ^'imposed by the limitations of available methodology.'* They note 
that 

Attempts to increase evaluative influence which focus 
on a few. of these factors in isolation and which do not 
recognize the highly complex and interactive system of 
forces constraining evaluator adtiuity-ai^likely to 
fail to alter the overall effect^s of the system (p,23). 

. . 

Their proposed solution would alter the institutional system for govemmentally 
funded external evaluations^ ^ ■ • ^ 
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- " ' '^> ■ i 

r ^ Under the new^&tem, ''oparatlonal feedback" provided by locally sponsored 

formative evaluations would create "direct and short-grange Improvements in 

\ - ' ■ * 

Specific programs/* More diffuse, long*range feeclback, focusing on the con- 

ceptual use of ^valuation results (wHat they call the "appreciative" use), 

would be provided by *'issue area^ evaluators" hired directly by funding agencies. 

\ 

The benefits of thfe proposed system would derive first f rtim the Involvement of 
the fojfmatlve Valuation contractors In the political miliTeu of the programs 
they evaluate, while at the same time relieving them of agency-imposed norms- 
Panels of Issue area evaluators, representing both policymakers and evaluators, 
would, on the o^er hand, be freed of political and funding constraints, - 
focusing instead oiy^road public policy issues over a period of time. 
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♦ 

"Weiss> C, H, Conceptual Issues In measuring the utilization of resiearbh 
and levaluatlQi; . A paper presented at the annual meeting of the Evaluation 
Research SQciety> Minneapolis > 1979, 

■ . ' 1 

'Although research utilizatlon'sounds like a '^straightforward and obvious 
event/* in practice the Interaction between evaluation results and the acts of 
program managers can be varied anil complex. Weiss presents fiv^ general cases 
where this complexity may create problens: 1) *valuaj:lon studies that do not 
produce clearcut answefrs; 2) studies that provide conflicting results; 3) studies 
whose results caSnot be implemented for whatever reason; 4) studies where evaluation 
results are only o^e type ^f Information available to the decision-maker; and 5) 
studies where managfers have survival rather than effectiveness on their minds. 
The conventional approach to studying utilization Is limited both because 
It has typically 'focused on Instrumental uses, ji,,e. on the explicit use of 
certain evaluatiort results* and because It has implicitly assumejd that use is 
^ood and non-use i)ad. This perspective is inappropriate glven'the complexities of 
organjfzational life, faking a broader view of u$e> however* what evaluation 
researcTi can do is to elaborate the context In which evaluation decisions are 
made and toj>rovide> dnce numerous studies are available* generalizations at^out 

the theory underlying a program. Researchers have demonstrated empirically 

■ ^ . ^1 

that the conceptual use of evaluation Is 

more prevalent than the rufliental use described above, Weiss feels that a 
use continuum exists* with conceptual and Instrumental the extremes, and most 
evaluation uses of interest lying somewhere in the middle, 

'The d"efl6lt;Lon of a "use-* Is clearly a centlral Issue for the study of 
evaluation uses. Specifically* researchers 'must determine whatsis used^ how 
directly the study Itself is used> who uses^lt> how many people use lt.> how 
'immediately It Is u$ed> and how much effect Is required to count as a use. An 
accepted and standard deflnltlorf will be of use in developing a cumulative 
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understanding of evaluation use in .the organizatlcxnal context-^ 

Given an agreed-upon definition of use, researchers also need to eicamine 
the approaches to their subject to determine which approach makes the most 
sense fpr a glv6n question. Weiss discusses fo'ur possibilities: 1) following 
the effects of selecte^,studles on subsequent decisions; 2) talking with 
prospective users of studlas; 3) examining the ways an Issue has beega treated; 
and. 4) studylng*J:he effects of research-and evaluation op selected organizations. 
Appropriate questions and metliods of study differ for each of these approaches, 
as do ^he associated limitations. 
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Wise, R. I . What wit know about the decision-maker and decision settings . 
Paper presented at the ^nual Meeting of the Ainerlcap Educational Research 
Association, Toronto, 1978. 

In this paper Wise places the blame for the underutlllzatlon of ^evaluation 
results on the evaluation communlty*s normative view of how decisions ought to 
be made, i,e. the Improper assumption that decision-makers in the r^al w6rld 
^should act according to a Restrictive definition of rationality. E/evlewlng, 
f^.ve different models of the strategies decision-makers use. Wise writes that 
the disjointed Incrementallsm model^descrlbed by Braybrook and Llndbloom Is our 
'*best current answer to the question of what we^ know about decision-makers and 
decision making** (p, 21). in this model, decision-makers are sean as focusing 

7 

^n a never-endlfig series of Immediate problems, making continual' but relatively 
small changes In the status Quo. 

Wise presents three Implications of this model for evaluators, Tlrst, 
evaluators need to address all actors involved In progrdrms> I.e. stakeholders* 
Influencersj and adoptors. Second, to influence decision-making* evaluators 
must involve themselves In immediate .programs and proposals for dealing with 
such problems. Third, they must use Information and argunients that reflect an 

1 

understanding of both the problem and the alternative proposals. He concludes* 
"The direction of these alternative premises Is away from an Ideology based on the 
notion of a client with a need for information to make a decision at a particular 
time,'* suggesting that If an evaluation utilization problem exists* It may well 
exist in the prescriptive view of evaluators unaple to "see their Information 
being used In the incrementalis^of real-world d^ciaJto^^aking^ (pp, 23-24) 
Wise's view provides additional theoretical si^P^rt for-the alternative view of 
utilization given In Alkln* DalUak, and W^iit^ (1979) . 



ERIC 



